Skip to content

Report stats span collapses over telemetry (client-side stats) - #12070

Open
dougqh wants to merge 6 commits into
masterfrom
dougqh/stats-collapsed-spans-telemetry
Open

Report stats span collapses over telemetry (client-side stats)#12070
dougqh wants to merge 6 commits into
masterfrom
dougqh/stats-collapsed-spans-telemetry

Conversation

@dougqh

@dougqh dougqh commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Report client-side trace-stats span collapses over telemetry as a new stats.collapsed_spans counter (namespace tracers), mirroring the existing statsd metric datadog.tracer.stats.collapsed_spans.

Motivation

Required by the cardinality limits & span-derived tags RFCs

The stats engine already counts every span collapse — cardinality-limit collapses, length ("oversized") collapses, and whole-key table drops — and emits them to statsd. But that path is invisible when no dogstatsd sink is configured: the aggregator's HealthMetrics is HealthMetrics.NO_OP in that case (CoreTracer.java:735), so the counts are simply dropped. Telemetry is always wired, so surfacing the same signal there makes cardinality-collapse visibility independent of statsd.

Additional Notes

Implementation Details

  • New StatsMetrics holder (in internal-api, alongside BaggageMetrics): a per-reason AtomicLong counter in a ConcurrentHashMap, drained by CoreMetricCollector as a count metric — same pattern as the baggage counters.
  • Fed directly at each collapse site (CoreHandlers, PeerTagSchema, AdditionalTagsSchema, and the whole-key drop in Aggregator), right next to the existing HealthMetrics call — deliberately not hooked inside TracerHealthMetrics, so it fires even when statsd is off.
  • Tagged by collapse reason, matching the statsd tags: collapsed:<field>, collapsed:peer_tags, collapsed:additional_metric_tags, oversized:additional_metric_tags, collapsed:whole_key.

Cardinality safety

The per-reason map is dynamic, but the reason set is bounded by construction (9 core fields + peer_tags + the two additional-tags reasons + whole_key) — the cardinality limits this metric reports on are the very thing that bound it. So the map itself cannot blow up.

Testing

  • StatsMetricsTest (JUnit 5): accumulation, drain-delta semantics, per-reason isolation, non-positive no-op.
  • AdditionalTagsSchemaTest: a new case asserting the reset site feeds the telemetry counter (not just statsd).

Stacked on #11402 (client-side additional metric tags) — shares the schema/handler reset sites introduced there.

🤖 Generated with Claude Code

@dougqh dougqh added comp: metrics Metrics tag: ai generated Largely based on code generated by an AI or LLM type: enhancement labels Jul 24, 2026
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Jul 24, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 80.49%
Overall Coverage: 41.97% (-15.73%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 869b8b7 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.96 s 13.99 s [-0.9%; +0.4%] (no difference)
startup:insecure-bank:tracing:Agent 12.89 s 13.04 s [-1.9%; -0.4%] (maybe better)
startup:petclinic:appsec:Agent 16.95 s 16.61 s [+0.9%; +3.2%] (maybe worse)
startup:petclinic:iast:Agent 16.84 s 16.99 s [-1.8%; -0.0%] (maybe better)
startup:petclinic:profiling:Agent 16.90 s 16.75 s [-0.6%; +2.4%] (no difference)
startup:petclinic:sca:Agent 16.95 s 16.84 s [-0.2%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 16.10 s 16.29 s [-2.1%; -0.3%] (maybe better)

Commit: 869b8b72 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@PerfectSlayer PerfectSlayer added type: feature Enhancements and improvements and removed type: enhancement labels Jul 27, 2026
@dougqh

dougqh commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@dougqh
dougqh changed the base branch from master to dougqh/metrics-arbitrary-tags July 27, 2026 13:27
@chatgpt-codex-connector

Copy link
Copy Markdown

💡 Codex Review

return tryMakeImmutableSet(configProvider.getList(TRACE_STATS_ADDITIONAL_TAGS));

P2 Badge Snapshot the additional-tags configuration

When DD_TRACE_STATS_ADDITIONAL_TAGS is enabled, this getter re-queries ConfigProvider instead of returning an immutable value captured by the Config constructor. Consequently, mutable sources such as system properties can change this setting after configuration initialization, and the new setting is also absent from Config.toString() startup diagnostics. Store it in a final field during construction and include it in the configuration summary, as required for newly added configurations.

AGENTS.md reference: AGENTS.md:L37-L44



P2 Badge Preserve collapse deltas when the telemetry queue is full

When metricsQueue is full, this calls getValueAndReset() before discovering that offer() fails. TaggedCounter advances its previousCount during that call, so the rejected collapse delta is permanently lost instead of being retried on the next telemetry cycle; this is especially likely for these counters because they are collected after span and baggage metrics. Check queue capacity before draining or restore/re-add a rejected delta.


config.getTraceStatsCardinalityLimit(
"additional_tags", MetricCardinalityLimits.ADDITIONAL_TAG_VALUE),

P2 Badge Reject unsafe additional-tag cardinality limits

When additional tags are enabled and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT is set above 1 << 29, this value is passed directly to TagCardinalityHandler, whose constructor throws IllegalArgumentException; large values below that guard can instead attempt multi-gigabyte array allocations. Because this happens while constructing ClientStatsAggregator, a malformed setting can prevent tracer startup rather than falling back to the documented default. Validate a practical upper bound before constructing the schema.


if (previousPeerTagSchema != null) {
previousPeerTagSchema.resetHandlers(healthMetrics, cardinalityLimitReporter);
previousPeerTagSchema = null;

P2 Badge Retain retired schemas until producers release them

If a producer reads the old peer-tag schema and is then paused for more than one reporting interval—for example during a long scheduling or GC pause—this reset discards previousPeerTagSchema before that producer queues its trace. When the delayed snapshots are eventually consumed, they update the retired schema's handlers, but no later cycle resets that schema, so any resulting peer-tag collapse counts are silently omitted from health metrics and the new telemetry counter. Retain retired schemas until delayed producers can no longer reference them, rather than for a fixed single cycle.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@dougqh
dougqh force-pushed the dougqh/stats-collapsed-spans-telemetry branch from c1916aa to 302b882 Compare July 27, 2026 14:58
@dougqh
dougqh marked this pull request as ready for review July 27, 2026 16:00
@dougqh
dougqh requested a review from a team as a code owner July 27, 2026 16:00

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The new collapse counters preserve per-reason accumulation and delta-drain behavior under repeated updates, including non-positive counts; the guarded collector path also leaves pending deltas intact when telemetry capacity is full. No diff-only behavioral regression was identified. Full Gradle tests could not start because this sandbox lacks the required Java 25 toolchain.

Was this helpful? React 👍 or 👎

📊 Validated against 5 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit 545e3f0 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 545e3f03c9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal-api/src/main/java/datadog/trace/api/telemetry/CoreMetricCollector.java Outdated
@dougqh

dougqh commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @codex. Reconciling against the current branch — this review was posted against c1916aa269, before the rebase onto the updated #11402 base and the follow-up commits:

  1. Snapshot the additional-tags configuration (Config) — ✅ Fixed on base PR Add span-derived primary tags (CSS v1.3.0) #11402: traceStatsAdditionalTags is now a final field resolved in the constructor, returned directly by the getter, and included in toString() (f5b083bd80).
  2. Preserve collapse deltas when the telemetry queue is full (CoreMetricCollector) — ✅ Fixed on this branch: prepareMetrics() now checks remainingCapacity() before getValueAndReset() and stops the stats drain early, so a full queue no longer resets a delta it can't enqueue; untouched counters are collected on the next cycle (545e3f03c9). The same guard for the pre-existing span/baggage loops is split out into Guard core-metric telemetry drain against queue-full delta loss #12081.
  3. Reject unsafe additional-tag cardinality limits (ClientStatsAggregator) — ✅ Fixed on base PR Add span-derived primary tags (CSS v1.3.0) #11402: DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT flows through Config.getTraceStatsCardinalityLimit(...), which now rejects values above MAX_TRACE_STATS_CARDINALITY_LIMIT (1<<16) and falls back to the default rather than allocating oversized arrays or throwing during startup (4188b43471).
  4. Retain retired schemas until producers release them (ClientStatsAggregator) — Known limitation, accepted by design. The retired previousPeerTagSchema is held for one reset cycle; the failure mode needs a peer-tag schema change (config / feature-discovery driven, rare) to coincide with a producer paused for more than a full reporting interval (~10s), and it affects only block-telemetry counts, never trace data. Retaining schemas until "no producer can reference them" would require producer-side liveness tracking we don't want on the hot path. Treating this as a pre-existing config-update race, not a Report stats span collapses over telemetry (client-side stats) #12070 regression.

Items 1–3 are resolved; item 4 is a documented accepted limitation.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@dougqh dougqh changed the title Report stats span collapses over telemetry (stats.collapsed_spans) Report stats span collapses over telemetry (client-side stats) Jul 27, 2026
Base automatically changed from dougqh/metrics-arbitrary-tags to master July 27, 2026 21:06
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot requested a review from a team as a code owner July 27, 2026 21:06
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot requested review from amarziali and removed request for a team July 27, 2026 21:06
dougqh and others added 4 commits July 28, 2026 09:08
The client-side trace-stats engine already emits a statsd metric
(datadog.tracer.stats.collapsed_spans) whenever spans collapse under a
cardinality/length limit or a whole-key table-drop. That signal is invisible
when no dogstatsd sink is configured, since the aggregator's HealthMetrics is
NO_OP in that case.

Add a parallel telemetry counter, stats.collapsed_spans (namespace tracers),
tagged by collapse reason (collapsed:<field>, collapsed:peer_tags,
collapsed:additional_metric_tags, oversized:additional_metric_tags,
collapsed:whole_key). It is fed directly at each reset/drop site alongside the
existing HealthMetrics call, so it is independent of the statsd sink, and
drained by CoreMetricCollector like the baggage counters. The reason set is
bounded by construction, so the dynamic per-reason map cannot itself grow
unboundedly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ookup

The collapsed:whole_key counter is incremented once per dropped span on the
aggregator thread (Aggregator, findOrInsert == null branch), unlike every other
collapse reason which is batched once per reporting cycle. Under a cardinality
explosion the aggregate table pins at cap and every arriving span is dropped,
turning that path hot on an already-bottlenecked thread -- yet it went through a
ConcurrentHashMap.computeIfAbsent lookup on StatsMetrics per span.

Pre-create the whole_key TaggedCounter at construction (still registered in the
reason map so the telemetry drain picks it up unchanged) and add a dedicated
onWholeKeyCollapse() that increments the cached reference directly. The hot path
no longer touches the map; the batched reasons still route through
onCollapsedSpans as before.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
getValueAndReset() resets a counter's delta baseline, so calling it before the
offer() succeeds means a full metricsQueue silently drops that delta forever --
exactly when it matters most (telemetry backing up because the agent is
unreachable). Check remainingCapacity() before reading any counter and stop the
drain early instead; the untouched counters are collected on the next cycle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/stats-collapsed-spans-telemetry branch from 545e3f0 to 6431932 Compare July 28, 2026 13:13

@bric3 bric3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I have a concern on the method names. The code explicitly mentions it's mirroring on the same signal, yet the method names are different.

Moreover the generated doc is still awkward, in particular in StatsMetrics.java.

Comment on lines 127 to +128
healthMetrics.onTagCardinalityBlocked(COLLAPSED_STATSD_TAG, totalCollapsed);
StatsMetrics.getInstance().onCollapsedSpans(COLLAPSED_STATSD_TAG[0], totalCollapsed);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Why method are named differently, this raises question when reading the code, yet, it seems the sgtats are emitted on the same "conditions".

I suggest aligning method names, unless there's a valid reason not to.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point — juxtaposed like this they do read a little oddly. The names differ on purpose: each mirrors the vocabulary of the sink it feeds, and the two sinks are named independently.

  • HealthMetrics.onTagCardinalityBlocked(...) is one of HealthMetrics' onX event callbacks; that subsystem's vocabulary is "blocked" (the offending tag value is blocked/collapsed to the sentinel), and its statsd tags are collapsed:/oversized:.
  • StatsMetrics.onCollapsedSpans(...) mirrors the telemetry metric it increments, stats.collapsed_spans — the collapse reason rides in the tag rather than the method name.

So both fire under the same condition because they report the same underlying event to two sinks, but renaming either to match the other would make that method name diverge from the name of the thing it actually reports. I'd rather keep each aligned to its own metric/event name.

Open to a different split if you feel strongly — but I don't think there's a single verb that reads naturally against both stats.collapsed_spans and the health *_blocked family at once.

dougqh and others added 2 commits July 28, 2026 12:27
…istry

The span-metric registry holds one entry per instrumentation name and can
exceed the 1024-entry telemetry queue on its own. Draining it before the small,
fixed set of client-side stats collapse counters could fill the queue and
starve those counters indefinitely. Collect them first so they are always
emitted; the pre-read capacity guard still protects their delta baseline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
StatsMetrics is a process-wide singleton; these tests left per-reason deltas
behind, leaking stats.collapsed_spans metrics into CoreMetricCollectorTest when
they shared a Gradle worker and making its exact span-metric count assertion
order-dependent. Reset every counter in @AfterEach.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: metrics Metrics tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants